The present invention relates to neural networks, and more particularly, to a chaotic recurrent neural network and a learning algorithm therefor.
Recently, various research efforts into neural networks have been actively progressed. These efforts include studies of a neural network model having a recurrent connection where time-variant input and output signals can be processed. Also, a back-propagation through time (BPTT) algorithm is widely used to convert the recurrent neural network into a multilayer feed-forward network for learning, and is so named (back-propagation) due to the reversed learning process employed. Here, a discrete-time model of the recurrent neural network is spatially unfolded to realize the multilayer feed-forward network.
In the above BPTT algorithm, the number of calculations equals the square of the cell number. An updated trajectory value has to be temporarily stored as the dynamics of a network are calculated over time (from time "O" to time "T") and then the error has to be reversely calculated (from time "T" to time "O"). As the learning algorithm of a recurrent neural network which overcomes the above drawbacks in calculation, a finite time learning algorithm, where the neural network is operated after a connecting weight is fixed during a predetermined time period, thereby reducing most of the system error in a given time period, has been suggested. The recurrent neural network is classified according to whether the connecting weight is regarded as a time-variant function or a time-invariant value which is constant during a predetermined time period of the network operations. Here, the finite time learning algorithm is the latter (time-variant) value for minimizing error after termination of the time period.
A neural network is composed of N neurons satisfying the following dynamic equations. ##EQU2## Here, i is a natural number from one to N; Y.sub.i (t) is the output of the "i"th neuron; f.sub.i is the output function of a neuron; .gamma..sub.i is a time delay constant; and W.sub.ij is a connecting weight of the "j"th neuron, which is a time-invariant value. Also, the network has a bias signal a(t) externally provided at time t, as a time-variant function.
FIG. 1 shows the conventional recurrent neural network model.
In this case, the network operates for a given time period according to a predetermined initial condition and an external input. During network operation, the connecting weight between neurons is fixed and the error is accumulated over time. The learning function of a network is defined as the total error of the network and is calculated during a predetermined time period as follows. ##EQU3## Here, X.sub.i t.vertline.W) is the output of the "i"th neuron of a network at time t, for a fixed connecting weight matrix W which is calculated from equations (1) and (2); and Q.sub.i (t) is a given time-variant teacher signal. In this case, a steepest descent method is used as a weight correction rule and the connecting weight correction amount calculated by introducing a Lagrange multiplier L.sub.i (t) is as follows. ##EQU4## Here, .eta. is a positive constant.
This method has been adopted for handwritten numeral recognition so that time sequential data can be recognized and estimated.
The research into neural computers for imitating the structure of a cerebral neural network and the information processing mechanism thereof began with a mathematical neural model based on the digital characteristics of an active potential pulse. However, recently, interest in the analog characteristics of a neuron has gradually increased. That is, the extraordinarily dynamic behavior of a cerebral nervous system exhibits a response characteristic called "chaos" which cannot be described by the conventional neuron model. Therefore, in a recently suggested chaos neuron model, the non-linear analog characteristics of the brain are emphasized. This model performs an analog correction of the response characteristic so that the qualitative description of a chaos response characteristic is possible.
The chaos neuron model with respect to one input is expressed as: ##EQU5## where X(t) is the output of a neuron at time t and corresponds to the peak value of a neural pulse (0.ltoreq.X(t).ltoreq.1), f is an output function of the neuron, A(t) is the magnitude of an input stimulus at time t, .alpha. is a non-negative parameter (.alpha..gtoreq.0), k is a refractory time attenuation constant (0.ltoreq.k.ltoreq.1), g is a function showing the relation between the output of a neuron and the refractory in response to the next stimulus (hereinafter, supposing that identical function g(x) equals x, for simplification), and e is a threshold value.
Supposing a time-spacial summation, where a previous value is added to the current value and attenuated over time (here, attenuation constant is k), as the same with the refractory, the input and output characteristics of a neuron can be expressed by the following equation (6). ##EQU6## Here, X.sub.i (t+1) is the output of the "i"th neuron at time t+1; V.sub.ij is a synapse connection coefficient from the "j"th external input to the "i"th neuron; A.sub.j (t) is the magnitude of the "j"th external input at time t, W.sub.ij is a synapse connection coefficient of the feedback input from the "j"th neuron to the "i"th neuron; and .theta. is the threshold of the "i"th neuron.
Here, the discrete-time dynamics of a chaotic neuron are expressed by the following simple differential equation. ##EQU7##
The neural network expressed as equations (7), (8) and (9) is called a chaotic neural network. When the chaotic neural network is applied to an associative memory model, the network is not stably converged to one memory pattern which is the closest to a current input and dynamically oscillates over various memory patterns according to the values of the parameters.
After the chaotic neural network model was suggested, the analysis of a constant input regardless of time is proceeded, by applying the model to an operation memory device. However, so far, there has been nothing reported on a chaotic neural network employing a time-variant input and output. In order to observe the possibility for the learning of a chaotic neural network using this time-variant input and output and the application thereof, the chaotic recurrent neural network is composed of chaotic neurons and a finite time learning algorithm improved by properly correcting the finite time learning algorithm for general recurrent neural network is suggested. Also, the effects of the refractory parameters of chaotic neural network in the learning of the chaotic recurrent neural network according to this method will be analyzed.
However, since the suggested finite time learning algorithm is a kind of reversed recurrent neural network, the same problems as those shown in the multi-neutral network using the reversed method are generated when the method is applied. The problems are local minima and a long learning time.
FIG. 2 is a graph showing total error with respect to learning iteration of the recurrent neural network shown in FIG. 1. Here, it can be seen that as the number of iterations increases, the accumulative error is not largely corrected and its slope is relatively steep as it approaches the teacher signal. To overcome this problem, the present invention adopts a chaotic neural network. The chaotic neural network is induced from a discrete-time neuron model. The chaotic neural network exhibits both a periodical response (which cannot be realized by the conventional neuron model) and a chaotic response.